This notebook contains the key visualizations for the Task Mapping
paper.
Very useful decision boundary plotting code from: https://mhahsler.github.io/Introduction_to_Data_Mining_R_Examples/book/classification-alternative-techniques.html#k-nearest-neighbors
decisionplot <- function(model, data, class_var,
predict_type = c("class", "prob"), resolution = 5 * 75) {
# resolution is set to 75 dpi if the image is rendered 5 inces wide.
y <- data %>% pull(class_var)
x <- data %>% dplyr::select(-all_of(class_var))
# resubstitution accuracy
prediction <- predict(model, x, type = predict_type[1])
# LDA returns a list
if(is.list(prediction)) prediction <- prediction$class
prediction <- factor(prediction, levels = levels(y))
cm <- confusionMatrix(data = prediction, reference = y)
acc <- cm$overall["Accuracy"]
# evaluate model on a grid
r <- sapply(x[, 1:2], range, na.rm = TRUE)
xs <- seq(r[1,1], r[2,1], length.out = resolution)
ys <- seq(r[1,2], r[2,2], length.out = resolution)
g <- cbind(rep(xs, each = resolution), rep(ys, time = resolution))
colnames(g) <- colnames(r)
g <- as_tibble(g)
### guess how to get class labels from predict
### (unfortunately not very consistent between models)
cl <- predict(model, g, type = predict_type[1])
# LDA returns a list
if(is.list(cl)) {
prob <- cl$posterior
cl <- cl$class
} else
try(prob <- predict(model, g, type = predict_type[2]))
# we visualize the difference in probability/score between the
# winning class and the second best class.
# don't use probability if predict for the classifier does not support it.
max_prob <- 1
try({
max_prob <- t(apply(prob, MARGIN = 1, sort, decreasing = TRUE))
max_prob <- max_prob[,1] - max_prob[,2]
}, silent = TRUE)
cl <- factor(cl, levels = levels(y))
g <- g %>% add_column(prediction = cl, probability = max_prob)
ggplot(g, mapping = aes_string(
x = colnames(g)[1],
y = colnames(g)[2])) +
geom_raster(mapping = aes(fill = prediction, alpha = probability)) +
geom_contour(mapping = aes(z = as.numeric(prediction)),
bins = length(levels(cl)), size = .5, color = "black") +
geom_point(data = data, mapping = aes_string(
x = colnames(data)[1],
y = colnames(data)[2],
shape = class_var), alpha = .7) +
scale_alpha_continuous(range = c(0,1), limits = c(0,1), guide = "none") +
labs(subtitle = paste("Training accuracy:", round(acc, 2)))
}
Load the Data
task_map <- read_csv('../task_map.csv')
Rows: 102 Columns: 25-- Column specification ----------------------------------------------------------------------------------------
Delimiter: ","
chr (1): task
dbl (24): Q1concept_behav, Q3type_1_planning, Q4type_2_generate, Q6type_5_cc, Q7type_7_battle, Q8type_8_perf...
i Use `spec()` to retrieve the full column specification for this data.
i Specify the column types or set `show_col_types = FALSE` to quiet this message.
Plot the Task Map and other Related Images
Draw the task map using PCA & clustering
First, run the PCA
set.seed(1)
pca <- task_map %>% #select(-continuous_questions) %>%
select(-task) %>%
prcomp(center = T)
# get optimal number of clusters -- "silhouette" method
fviz_nbclust(x = pca$x, FUNcluster = stats::kmeans, method = "silhouette") +
labs(subtitle = "Silhouette method")

# get optimal number of clusters
NbClust(data = pca$x, distance = "euclidean",
min.nc = 2, max.nc = 15, method = "kmeans")
*** : The Hubert index is a graphical method of determining the number of clusters.
In the plot of Hubert index, we seek a significant knee that corresponds to a
significant increase of the value of the measure i.e the significant peak in Hubert
index second differences plot.

*** : The D index is a graphical method of determining the number of clusters.
In the plot of D index, we seek a significant knee (the significant peak in Dindex
second differences plot) that corresponds to a significant increase of the value of
the measure.
*******************************************************************
* Among all indices:
* 9 proposed 2 as the best number of clusters
* 3 proposed 3 as the best number of clusters
* 2 proposed 4 as the best number of clusters
* 5 proposed 10 as the best number of clusters
* 2 proposed 13 as the best number of clusters
* 1 proposed 14 as the best number of clusters
* 2 proposed 15 as the best number of clusters
***** Conclusion *****
* According to the majority rule, the best number of clusters is 2
*******************************************************************
$All.index
KL CH Hartigan CCC Scott Marriot TrCovW TraceW Friedman Rubin Cindex DB
2 5.4470 61.4035 14.4335 0.9622 221.9375 164410.8392 48.5404 99.8470 6.7370 1.6140 0.4272 1.2541
3 0.9212 41.9264 15.6316 0.4562 397.8075 65962.2674 35.8229 87.2532 9.9607 1.8470 0.4592 1.7097
4 34.1211 37.1952 1.4591 1.7101 579.4047 19768.3559 25.6963 75.3550 15.5991 2.1386 0.4459 1.6280
5 0.1297 28.3839 4.4654 -0.1287 701.1324 9364.8251 25.6267 74.2495 20.5842 2.1705 0.4027 1.5794
6 0.2807 24.3914 12.9929 -0.8555 772.2602 6714.5189 23.1698 70.9819 19.9124 2.2704 0.3907 1.4941
7 5.0651 24.9798 3.4111 1.2775 931.4178 1919.7696 16.4934 62.5202 28.6105 2.5777 0.4335 1.5758
8 0.5516 22.4287 5.3286 0.0887 948.1492 2128.1093 15.0974 60.3532 28.9981 2.6702 0.4417 1.6543
9 0.2735 21.1760 19.9755 -0.1272 1097.9023 620.4146 13.6645 57.1155 37.1941 2.8216 0.4201 1.6851
10 6.9115 24.8159 3.5564 4.2723 1412.1301 35.1788 8.3850 47.0167 65.2931 3.4276 0.3675 1.4949
11 0.6885 23.2973 4.9020 3.6451 1501.7500 17.6801 8.4934 45.2669 71.7599 3.5601 0.3629 1.4095
12 1.7052 22.5157 3.1285 3.4910 1662.6670 4.3442 7.5335 42.9531 74.7842 3.7519 0.3648 1.2815
13 0.6085 21.3774 4.8517 2.8132 1725.7302 2.7474 7.0349 41.5101 77.8436 3.8823 0.3610 1.2427
14 1.0322 20.9439 4.8261 2.7635 1828.9448 1.1583 6.3322 39.3643 84.8649 4.0940 0.3264 1.3681
15 0.8228 20.6221 5.9833 2.8535 2033.4956 0.1790 6.2626 37.3177 89.8610 4.3185 0.3610 1.4006
Silhouette Duda Pseudot2 Beale Ratkowsky Ball Ptbiserial Frey McClain Dunn Hubert SDindex
2 0.3498 1.4321 -17.4995 -4.9369 0.0575 49.9235 0.6834 1.7824 0.5636 0.1982 0.0161 2.8083
3 0.2483 0.5136 28.4081 15.4698 0.0994 29.0844 0.5878 -0.0301 1.2183 0.2395 0.0164 3.3612
4 0.2538 3.6681 -25.4584 -10.7034 0.1083 18.8388 0.6390 1.5820 1.3200 0.2524 0.0197 3.0087
5 0.2249 1.6467 -5.1055 -5.5039 0.1270 14.8499 0.5845 0.2054 1.7697 0.2395 0.0175 3.1680
6 0.2210 4.8002 -30.0837 -11.4118 0.1315 11.8303 0.5888 1.3888 1.8554 0.2395 0.0174 2.9098
7 0.1657 1.0931 -1.5324 -1.3294 0.1362 8.9315 0.4876 -3.0782 3.1452 0.2207 0.0202 3.5619
8 0.1382 4.3481 -12.3202 -10.3596 0.1342 7.5441 0.4365 0.1626 3.9109 0.2223 0.0204 4.1061
9 0.1601 1.1020 -0.8329 -1.2969 0.1391 6.3462 0.4348 -0.1788 4.2569 0.2165 0.0209 3.7886
10 0.2342 2.6811 -22.5727 -9.8417 0.1416 4.7017 0.5062 -1.5738 3.6719 0.2190 0.0228 3.7292
11 0.2249 0.6907 4.0296 6.8452 0.1423 4.1152 0.4794 -0.1361 4.0758 0.1755 0.0232 3.5309
12 0.2415 1.0628 -0.2953 -0.7449 0.1468 3.5794 0.4938 0.3309 3.9643 0.1818 0.0232 3.4085
13 0.2437 0.8068 5.0290 3.8442 0.1461 3.1931 0.4908 0.2872 4.0520 0.1818 0.0235 3.5236
14 0.2070 2.3198 -3.9826 -6.3786 0.1451 2.8117 0.4649 1.0053 4.8952 0.1818 0.0247 3.4845
15 0.2032 1.1460 -2.2931 -1.9996 0.1477 2.4878 0.4335 0.2185 5.7441 0.2738 0.0251 3.9810
Dindex SDbw
2 0.9422 0.4544
3 0.8797 0.3788
4 0.8187 0.3184
5 0.8118 0.3386
6 0.7922 0.3131
7 0.7474 0.2751
8 0.7342 0.2539
9 0.7112 0.2516
10 0.6467 0.2004
11 0.6319 0.1808
12 0.6165 0.1643
13 0.6064 0.1555
14 0.5904 0.1560
15 0.5743 0.1588
$All.CriticalValues
CritValue_Duda CritValue_PseudoT2 Fvalue_Beale
2 0.8262 12.2032 1
3 0.8221 6.4903 0
4 0.6708 17.1766 1
5 0.6303 7.6260 1
6 0.6524 20.2431 1
7 0.7390 6.3565 1
8 0.6028 10.5440 1
9 0.6303 5.2795 1
10 0.7465 12.2272 1
11 0.7113 3.6520 0
12 0.5674 3.8122 1
13 0.7841 5.7810 0
14 0.5195 6.4755 1
15 0.7465 6.1136 1
$Best.nc
KL CH Hartigan CCC Scott Marriot TrCovW TraceW Friedman Rubin Cindex
Number_clusters 4.0000 2.0000 10.000 10.0000 10.0000 3.00 3.0000 4.0000 10.000 10.0000 14.0000
Value_Index 34.1211 61.4035 16.419 4.2723 314.2278 52254.66 12.7175 10.7927 28.099 -0.4735 0.3264
DB Silhouette Duda PseudoT2 Beale Ratkowsky Ball PtBiserial Frey McClain Dunn
Number_clusters 13.0000 2.0000 2.0000 2.0000 2.0000 15.0000 3.0000 2.0000 2.0000 2.0000 15.0000
Value_Index 1.2427 0.3498 1.4321 -17.4995 -4.9369 0.1477 20.8391 0.6834 1.7824 0.5636 0.2738
Hubert SDindex Dindex SDbw
Number_clusters 0 2.0000 0 13.0000
Value_Index 0 2.8083 0 0.1555
$Best.partition
[1] 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2
[54] 2 1 1 2 2 2 2 2 2 1 2 1 2 2 2 2 2 2 1 1 1 2 2 2 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

kmeans_output <- pca$x %>%
kmeans(centers = 3, nstart = 100)
combined_data <- cbind(task_map,
pca$x, factor(kmeans_output$cluster)) %>%
rename(cluster = `factor(kmeans_output$cluster)`)
fviz_eig(pca)

Standard Task Map Image with All Labels
p <- combined_data %>%
ggplot(aes(
x = PC1,
y = PC2,
label = task,
fill = cluster
)) + geom_point() + geom_label(nudge_y = 0.1, size = 4) +
#+ , alpha=0.05) +
# highlights only the ones in the selected set
# geom_label(
# data = subset(combined_data, task %in% c("NASA Moon survival", "Desert survival")),
# aes(
# x = PC1,
# y = PC2,
# label = task ,
# fill = cluster
# ),
# nudge_y = 0.1,
# size = 2
# )
theme_light(base_size = 24)
p # show the plot

ggsave(plot = p, filename = '../task-map.png')
Saving 28 x 10 in image
Task Map Image Highlighting Specific Subsets (for Illustrative
Purposes)
# An illustrative set to display
display_set <- c('Writing story',
'Advertisement writing',
'Desert survival',
'NASA Moon survival',
'Ultimatum game (various versions)',
'Dictator game and its variants',
'Prisoner\'s Dilemma (various versions)',
'9 Dot Problem',
'Word construction from a subset of letters',
'Typing game',
'Ravens Matrices',
'Euclidean traveling salesperson'
)
# A set of the tasks that are most different
max_diff_set <- c('Shopping plan',
'Minimal Group Paradigm (study diversity)',
'9 Dot Problem',
'Whac-A-Mole',
'Bullard Houses',
'Putting food into categories',
'Checkers',
'Reproducing arts',
'Allocating resources to programs',
'Image rating',
'Arithmetic problem 2')
# A set of tasks that are the most similar
min_diff_set <- c('Arithmetic problem 1',
'Euclidean traveling salesperson',
'Abstract grid task',
'Mastermind',
'Logic Problem',
'Guessing the correlation',
'Random dot motion',
'Letters-to-numbers problems (cryptography)',
'Computer maze',
'Recall images',
'Recall stories')
# A set of tasks that illustrates opportunities to add new tasks
display_limitations_set <- c('Recall word lists',
'Hidden figures in a picture (Recall Task)',
'Recall images',
'Recall stories',
'Recall videos',
'Writing story',
'Advertisement writing')
graph_illustrative_plots <- function(displayset, filename){
p <- combined_data %>%
ggplot(aes(
x = PC1,
y = PC2,
#label = task,
#fill = cluster
)) + geom_point(aes(size = 4)) +
#geom_point(aes(color = cluster, size = 4)) +
#highlights only the ones in the selected set
geom_label(
data = subset(combined_data, task %in% displayset),
aes(
x = PC1,
y = PC2,
label = task
),
nudge_y = 0.15,
size = 4
) +
geom_point(data = subset(combined_data, task %in% displayset), aes(size = 4),
color = "firebrick1") +
theme_minimal(base_size = 18) + theme(legend.position = "none")
p
ggsave(plot = p, filename = filename, width = 14, height = 5)
}
#graph_illustrative_plots(display_limitations_set, '../images/task-map_with_new_task_opportunities_highlighted.png')
#graph_illustrative_plots(max_diff_set, '../images/task-map_with_max_diff_highlighted.png')
graph_illustrative_plots(min_diff_set, '../images/task-map_with_min_diff_highlighted.png')
Create a cool 3D version
plot_ly(
x = combined_data$PC1,
y = combined_data$PC2,
z = combined_data$PC3,
type = "scatter3d",
mode = "markers", # can use mode = "text"
text = combined_data$task ,
color = combined_data$cluster
)
Create synthetic dependent variable based on the clusters
tasks_with_dv <- subset(combined_data, task %in% max_diff_set) %>%
mutate(
synergy = as.factor(ifelse(cluster == 1 | cluster == 2, 1, 0))
)
combined_data <- combined_data %>%
mutate(
synergy = as.factor(ifelse(cluster == 1 | cluster == 2, 1, 0))
)
Fitting and Visualizing Models for the Task Map.
x <- combined_data %>% select(PC1, PC2, synergy, task)
train <- tasks_with_dv %>% select(PC1, PC2, synergy, task)
model <- train %>% svm(synergy ~ PC1 + PC2, data = ., kernel = "linear")
svmplot <- decisionplot(model, x, class_var = "synergy") +
geom_point(data = train, aes(x = PC1, y = PC2, shape = synergy), color = "darkolivegreen2", show.legend = F) +
geom_label(data = train, aes(label = task ), nudge_y = 0.1, size = 3) +
labs(title = "SVM (Linear Kernel)") +
theme_minimal(base_size = 12)
svmplot
ggsave('svmplot_synthetic_data.png', width = 12, height = 5)

model <- train %>% knn3(synergy ~ PC1 + PC2, data = ., k = 1)
knnplot <- decisionplot(model, x, class_var = "synergy") +
geom_point(data = train, aes(x = PC1, y = PC2, shape = synergy), color = "darkolivegreen2", show.legend = F) +
geom_label(data = train, aes(label = task ), nudge_y = 0.1, size = 3) +
labs(title = "kNN (1 neighbor)") +
theme_minimal(base_size = 12)
knnplot
ggsave('knnplot_synthetic_data.png', width = 12, height = 5)

LS0tCnRpdGxlOiAiUiBOb3RlYm9vayBmb3IgUGFwZXItUmVsYXRlZCBWaXN1YWxpemF0aW9ucyIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKVGhpcyBub3RlYm9vayBjb250YWlucyB0aGUga2V5IHZpc3VhbGl6YXRpb25zIGZvciB0aGUgVGFzayBNYXBwaW5nIHBhcGVyLgoKYGBge3J9CmxpYnJhcnkoZmFjdG9leHRyYSkKbGlicmFyeShOYkNsdXN0KQpsaWJyYXJ5KGNsdXN0ZXIpCmxpYnJhcnkocGxvdGx5KQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoY2FyZXQpICNmb3Iga25uCmxpYnJhcnkoZTEwNzEpICNmb3Igc3ZtCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkodGlkeXZlcnNlKQpgYGAKClZlcnkgdXNlZnVsIGRlY2lzaW9uIGJvdW5kYXJ5IHBsb3R0aW5nIGNvZGUgZnJvbTogaHR0cHM6Ly9taGFoc2xlci5naXRodWIuaW8vSW50cm9kdWN0aW9uX3RvX0RhdGFfTWluaW5nX1JfRXhhbXBsZXMvYm9vay9jbGFzc2lmaWNhdGlvbi1hbHRlcm5hdGl2ZS10ZWNobmlxdWVzLmh0bWwjay1uZWFyZXN0LW5laWdoYm9ycwpgYGB7ciBkZWNpc2lvbnBsb3R9CgpkZWNpc2lvbnBsb3QgPC0gZnVuY3Rpb24obW9kZWwsIGRhdGEsIGNsYXNzX3ZhciwgCiAgcHJlZGljdF90eXBlID0gYygiY2xhc3MiLCAicHJvYiIpLCByZXNvbHV0aW9uID0gNSAqIDc1KSB7CiAgIyByZXNvbHV0aW9uIGlzIHNldCB0byA3NSBkcGkgaWYgdGhlIGltYWdlIGlzIHJlbmRlcmVkICA1IGluY2VzIHdpZGUuIAogIAogIHkgPC0gZGF0YSAlPiUgcHVsbChjbGFzc192YXIpCiAgeCA8LSBkYXRhICU+JSBkcGx5cjo6c2VsZWN0KC1hbGxfb2YoY2xhc3NfdmFyKSkKICAKICAjIHJlc3Vic3RpdHV0aW9uIGFjY3VyYWN5CiAgcHJlZGljdGlvbiA8LSBwcmVkaWN0KG1vZGVsLCB4LCB0eXBlID0gcHJlZGljdF90eXBlWzFdKQogICMgTERBIHJldHVybnMgYSBsaXN0CiAgaWYoaXMubGlzdChwcmVkaWN0aW9uKSkgcHJlZGljdGlvbiA8LSBwcmVkaWN0aW9uJGNsYXNzCiAgcHJlZGljdGlvbiA8LSBmYWN0b3IocHJlZGljdGlvbiwgbGV2ZWxzID0gbGV2ZWxzKHkpKQogIAogIGNtIDwtIGNvbmZ1c2lvbk1hdHJpeChkYXRhID0gcHJlZGljdGlvbiwgcmVmZXJlbmNlID0geSkKICBhY2MgPC0gY20kb3ZlcmFsbFsiQWNjdXJhY3kiXQogIAogICMgZXZhbHVhdGUgbW9kZWwgb24gYSBncmlkCiAgciA8LSBzYXBwbHkoeFssIDE6Ml0sIHJhbmdlLCBuYS5ybSA9IFRSVUUpCiAgeHMgPC0gc2VxKHJbMSwxXSwgclsyLDFdLCBsZW5ndGgub3V0ID0gcmVzb2x1dGlvbikKICB5cyA8LSBzZXEoclsxLDJdLCByWzIsMl0sIGxlbmd0aC5vdXQgPSByZXNvbHV0aW9uKQogIGcgPC0gY2JpbmQocmVwKHhzLCBlYWNoID0gcmVzb2x1dGlvbiksIHJlcCh5cywgdGltZSA9IHJlc29sdXRpb24pKQogIGNvbG5hbWVzKGcpIDwtIGNvbG5hbWVzKHIpCiAgZyA8LSBhc190aWJibGUoZykKICAKICAjIyMgZ3Vlc3MgaG93IHRvIGdldCBjbGFzcyBsYWJlbHMgZnJvbSBwcmVkaWN0CiAgIyMjICh1bmZvcnR1bmF0ZWx5IG5vdCB2ZXJ5IGNvbnNpc3RlbnQgYmV0d2VlbiBtb2RlbHMpCiAgY2wgPC0gcHJlZGljdChtb2RlbCwgZywgdHlwZSA9IHByZWRpY3RfdHlwZVsxXSkKICAKICAjIExEQSByZXR1cm5zIGEgbGlzdAogIGlmKGlzLmxpc3QoY2wpKSB7IAogICAgcHJvYiA8LSBjbCRwb3N0ZXJpb3IKICAgIGNsIDwtIGNsJGNsYXNzCiAgfSBlbHNlCiAgICB0cnkocHJvYiA8LSBwcmVkaWN0KG1vZGVsLCBnLCB0eXBlID0gcHJlZGljdF90eXBlWzJdKSkKICAKICAjIHdlIHZpc3VhbGl6ZSB0aGUgZGlmZmVyZW5jZSBpbiBwcm9iYWJpbGl0eS9zY29yZSBiZXR3ZWVuIHRoZSAKICAjIHdpbm5pbmcgY2xhc3MgYW5kIHRoZSBzZWNvbmQgYmVzdCBjbGFzcy4KICAjIGRvbid0IHVzZSBwcm9iYWJpbGl0eSBpZiBwcmVkaWN0IGZvciB0aGUgY2xhc3NpZmllciBkb2VzIG5vdCBzdXBwb3J0IGl0LgogIG1heF9wcm9iIDwtIDEKICB0cnkoewogICAgbWF4X3Byb2IgPC0gdChhcHBseShwcm9iLCBNQVJHSU4gPSAxLCBzb3J0LCBkZWNyZWFzaW5nID0gVFJVRSkpCiAgICBtYXhfcHJvYiA8LSBtYXhfcHJvYlssMV0gLSBtYXhfcHJvYlssMl0KICB9LCBzaWxlbnQgPSBUUlVFKSAKICAKICBjbCA8LSBmYWN0b3IoY2wsIGxldmVscyA9IGxldmVscyh5KSkKICAKICBnIDwtIGcgJT4lIGFkZF9jb2x1bW4ocHJlZGljdGlvbiA9IGNsLCBwcm9iYWJpbGl0eSA9IG1heF9wcm9iKQogIAogIGdncGxvdChnLCBtYXBwaW5nID0gYWVzX3N0cmluZygKICAgIHggPSBjb2xuYW1lcyhnKVsxXSwKICAgIHkgPSBjb2xuYW1lcyhnKVsyXSkpICsKICAgIGdlb21fcmFzdGVyKG1hcHBpbmcgPSBhZXMoZmlsbCA9IHByZWRpY3Rpb24sIGFscGhhID0gcHJvYmFiaWxpdHkpKSArCiAgICAgZ2VvbV9jb250b3VyKG1hcHBpbmcgPSBhZXMoeiA9IGFzLm51bWVyaWMocHJlZGljdGlvbikpLCAKICAgICAgYmlucyA9IGxlbmd0aChsZXZlbHMoY2wpKSwgc2l6ZSA9IC41LCBjb2xvciA9ICJibGFjayIpICsKICAgIGdlb21fcG9pbnQoZGF0YSA9IGRhdGEsIG1hcHBpbmcgPSAgYWVzX3N0cmluZygKICAgICAgeCA9IGNvbG5hbWVzKGRhdGEpWzFdLAogICAgICB5ID0gY29sbmFtZXMoZGF0YSlbMl0sCiAgICAgIHNoYXBlID0gY2xhc3NfdmFyKSwgYWxwaGEgPSAuNykgKyAKICAgIHNjYWxlX2FscGhhX2NvbnRpbnVvdXMocmFuZ2UgPSBjKDAsMSksIGxpbWl0cyA9IGMoMCwxKSwgZ3VpZGUgPSAibm9uZSIpICsgIAogICAgbGFicyhzdWJ0aXRsZSA9IHBhc3RlKCJUcmFpbmluZyBhY2N1cmFjeToiLCByb3VuZChhY2MsIDIpKSkKfQpgYGAKCiMgTG9hZCB0aGUgRGF0YQpgYGB7cn0KdGFza19tYXAgPC0gcmVhZF9jc3YoJy4uL3Rhc2tfbWFwLmNzdicpCmBgYAoKIyBQbG90IHRoZSBUYXNrIE1hcCBhbmQgb3RoZXIgUmVsYXRlZCBJbWFnZXMKCkRyYXcgdGhlIHRhc2sgbWFwIHVzaW5nIFBDQSAmIGNsdXN0ZXJpbmcKCkZpcnN0LCBydW4gdGhlIFBDQQpgYGB7ciwgZmlnLndpZHRoPTE2LCBmaWcuaGVpZ2h0PTV9CnNldC5zZWVkKDEpCgpwY2EgPC0gdGFza19tYXAgJT4lICNzZWxlY3QoLWNvbnRpbnVvdXNfcXVlc3Rpb25zKSAlPiUKICBzZWxlY3QoLXRhc2spICU+JQogIHByY29tcChjZW50ZXIgPSBUKQoKIyBnZXQgb3B0aW1hbCBudW1iZXIgb2YgY2x1c3RlcnMgLS0gInNpbGhvdWV0dGUiIG1ldGhvZApmdml6X25iY2x1c3QoeCA9IHBjYSR4LCBGVU5jbHVzdGVyID0gc3RhdHM6OmttZWFucywgbWV0aG9kID0gInNpbGhvdWV0dGUiKSArCiAgbGFicyhzdWJ0aXRsZSA9ICJTaWxob3VldHRlIG1ldGhvZCIpCgojIGdldCBvcHRpbWFsIG51bWJlciBvZiBjbHVzdGVycwpOYkNsdXN0KGRhdGEgPSBwY2EkeCwgZGlzdGFuY2UgPSAiZXVjbGlkZWFuIiwKICAgICAgICBtaW4ubmMgPSAyLCBtYXgubmMgPSAxNSwgbWV0aG9kID0gImttZWFucyIpCgprbWVhbnNfb3V0cHV0IDwtIHBjYSR4ICU+JSAKICBrbWVhbnMoY2VudGVycyA9IDMsIG5zdGFydCA9IDEwMCkKCmNvbWJpbmVkX2RhdGEgPC0gY2JpbmQodGFza19tYXAsCiAgICAgIHBjYSR4LCBmYWN0b3Ioa21lYW5zX291dHB1dCRjbHVzdGVyKSkgJT4lCiAgcmVuYW1lKGNsdXN0ZXIgPSBgZmFjdG9yKGttZWFuc19vdXRwdXQkY2x1c3RlcilgKQoKZnZpel9laWcocGNhKQpgYGAKClN0YW5kYXJkIFRhc2sgTWFwIEltYWdlIHdpdGggQWxsIExhYmVscwpgYGB7ciwgZmlnLndpZHRoPTE0LCBmaWcuaGVpZ2h0PTV9CnAgPC0gY29tYmluZWRfZGF0YSAlPiUKICBnZ3Bsb3QoYWVzKAogICAgeCA9IFBDMSwKICAgIHkgPSBQQzIsCiAgICBsYWJlbCA9IHRhc2ssCiAgICBmaWxsID0gY2x1c3RlcgogICkpICsgZ2VvbV9wb2ludCgpICsgZ2VvbV9sYWJlbChudWRnZV95ID0gMC4xLCBzaXplID0gNCkgKwogIAogICMrICwgYWxwaGE9MC4wNSkgKwojIGhpZ2hsaWdodHMgb25seSB0aGUgb25lcyBpbiB0aGUgc2VsZWN0ZWQgc2V0CiAgIyBnZW9tX2xhYmVsKAogICMgICBkYXRhID0gc3Vic2V0KGNvbWJpbmVkX2RhdGEsIHRhc2sgJWluJSBjKCJOQVNBIE1vb24gc3Vydml2YWwiLCAiRGVzZXJ0IHN1cnZpdmFsIikpLAogICMgICBhZXMoCiAgIyAgICAgeCA9IFBDMSwKICAjICAgICB5ID0gUEMyLAogICMgICAgIGxhYmVsID0gdGFzayAsCiAgIyAgICAgZmlsbCA9IGNsdXN0ZXIKICAjICAgKSwKICAjICAgbnVkZ2VfeSA9IDAuMSwKICAjICAgc2l6ZSA9IDIKICAjICkKIHRoZW1lX2xpZ2h0KGJhc2Vfc2l6ZSA9IDI0KQoKcCAjIHNob3cgdGhlIHBsb3QKCmdnc2F2ZShwbG90ID0gcCwgZmlsZW5hbWUgPSAnLi4vdGFzay1tYXAucG5nJykKYGBgCgpUYXNrIE1hcCBJbWFnZSBIaWdobGlnaHRpbmcgU3BlY2lmaWMgU3Vic2V0cyAoZm9yIElsbHVzdHJhdGl2ZSBQdXJwb3NlcykKYGBge3IsIGZpZy53aWR0aD0xNCwgZmlnLmhlaWdodD01fQojIEFuIGlsbHVzdHJhdGl2ZSBzZXQgdG8gZGlzcGxheQpkaXNwbGF5X3NldCA8LSBjKCdXcml0aW5nIHN0b3J5JywKICdBZHZlcnRpc2VtZW50IHdyaXRpbmcnLCAKICdEZXNlcnQgc3Vydml2YWwnLAogJ05BU0EgTW9vbiBzdXJ2aXZhbCcsCiAnVWx0aW1hdHVtIGdhbWUgKHZhcmlvdXMgdmVyc2lvbnMpJywKICdEaWN0YXRvciBnYW1lIGFuZCBpdHMgdmFyaWFudHMnLAogJ1ByaXNvbmVyXCdzIERpbGVtbWEgKHZhcmlvdXMgdmVyc2lvbnMpJywKICc5IERvdCBQcm9ibGVtJywKICdXb3JkIGNvbnN0cnVjdGlvbiBmcm9tIGEgc3Vic2V0IG9mIGxldHRlcnMnLAogJ1R5cGluZyBnYW1lJywKICdSYXZlbnMgTWF0cmljZXMnLAogJ0V1Y2xpZGVhbiB0cmF2ZWxpbmcgc2FsZXNwZXJzb24nCiApCgojIEEgc2V0IG9mIHRoZSB0YXNrcyB0aGF0IGFyZSBtb3N0IGRpZmZlcmVudAptYXhfZGlmZl9zZXQgPC0gYygnU2hvcHBpbmcgcGxhbicsCiAnTWluaW1hbCBHcm91cCBQYXJhZGlnbSAoc3R1ZHkgZGl2ZXJzaXR5KScsCiAnOSBEb3QgUHJvYmxlbScsCiAnV2hhYy1BLU1vbGUnLAogJ0J1bGxhcmQgSG91c2VzJywKICdQdXR0aW5nIGZvb2QgaW50byBjYXRlZ29yaWVzJywKICdDaGVja2VycycsCiAnUmVwcm9kdWNpbmcgYXJ0cycsCiAnQWxsb2NhdGluZyByZXNvdXJjZXMgdG8gcHJvZ3JhbXMnLAogJ0ltYWdlIHJhdGluZycsCiAnQXJpdGhtZXRpYyBwcm9ibGVtIDInKQoKIyBBIHNldCBvZiB0YXNrcyB0aGF0IGFyZSB0aGUgbW9zdCBzaW1pbGFyCm1pbl9kaWZmX3NldCA8LSBjKCdBcml0aG1ldGljIHByb2JsZW0gMScsCiAnRXVjbGlkZWFuIHRyYXZlbGluZyBzYWxlc3BlcnNvbicsCiAnQWJzdHJhY3QgZ3JpZCB0YXNrJywKICdNYXN0ZXJtaW5kJywKICdMb2dpYyBQcm9ibGVtJywKICdHdWVzc2luZyB0aGUgY29ycmVsYXRpb24nLAogJ1JhbmRvbSBkb3QgbW90aW9uJywKICdMZXR0ZXJzLXRvLW51bWJlcnMgcHJvYmxlbXMgKGNyeXB0b2dyYXBoeSknLAogJ0NvbXB1dGVyIG1hemUnLAogJ1JlY2FsbCBpbWFnZXMnLAogJ1JlY2FsbCBzdG9yaWVzJykKCiMgQSBzZXQgb2YgdGFza3MgdGhhdCBpbGx1c3RyYXRlcyBvcHBvcnR1bml0aWVzIHRvIGFkZCBuZXcgdGFza3MKZGlzcGxheV9saW1pdGF0aW9uc19zZXQgPC0gYygnUmVjYWxsIHdvcmQgbGlzdHMnLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICdIaWRkZW4gZmlndXJlcyBpbiBhIHBpY3R1cmUgKFJlY2FsbCBUYXNrKScsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ1JlY2FsbCBpbWFnZXMnLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICdSZWNhbGwgc3RvcmllcycsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ1JlY2FsbCB2aWRlb3MnLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICdXcml0aW5nIHN0b3J5JywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnQWR2ZXJ0aXNlbWVudCB3cml0aW5nJykKCgpncmFwaF9pbGx1c3RyYXRpdmVfcGxvdHMgPC0gZnVuY3Rpb24oZGlzcGxheXNldCwgZmlsZW5hbWUpewogIHAgPC0gY29tYmluZWRfZGF0YSAlPiUKICBnZ3Bsb3QoYWVzKAogICAgeCA9IFBDMSwKICAgIHkgPSBQQzIsCiAgICAjbGFiZWwgPSB0YXNrLAogICAgI2ZpbGwgPSBjbHVzdGVyCiAgICApKSArIGdlb21fcG9pbnQoYWVzKHNpemUgPSA0KSkgKwogICNnZW9tX3BvaW50KGFlcyhjb2xvciA9IGNsdXN0ZXIsIHNpemUgPSA0KSkgKwojaGlnaGxpZ2h0cyBvbmx5IHRoZSBvbmVzIGluIHRoZSBzZWxlY3RlZCBzZXQKZ2VvbV9wb2ludChkYXRhID0gc3Vic2V0KGNvbWJpbmVkX2RhdGEsIHRhc2sgJWluJSBkaXNwbGF5c2V0KSwgYWVzKHNpemUgPSA0KSwKICAgICAgICAgICBjb2xvciA9ICJmaXJlYnJpY2sxIikgKwpnZW9tX2xhYmVsKAogIGRhdGEgPSBzdWJzZXQoY29tYmluZWRfZGF0YSwgdGFzayAlaW4lIGRpc3BsYXlzZXQpLAogIGFlcygKICAgIHggPSBQQzEsCiAgICB5ID0gUEMyLAogICAgbGFiZWwgPSB0YXNrCiAgKSwKICBudWRnZV95ID0gMC4xNSwKICBzaXplID0gNAopICsKIHRoZW1lX21pbmltYWwoYmFzZV9zaXplID0gMTgpICsgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gIm5vbmUiKSAgCgpwCgpnZ3NhdmUocGxvdCA9IHAsIGZpbGVuYW1lID0gZmlsZW5hbWUsIHdpZHRoID0gMTQsIGhlaWdodCA9IDUpCn0KYGBgCgpgYGB7ciwgZmlnLndpZHRoPTE0LCBmaWcuaGVpZ2h0PTV9CmdyYXBoX2lsbHVzdHJhdGl2ZV9wbG90cyhkaXNwbGF5X2xpbWl0YXRpb25zX3NldCwgJy4uL2ltYWdlcy90YXNrLW1hcF93aXRoX25ld190YXNrX29wcG9ydHVuaXRpZXNfaGlnaGxpZ2h0ZWQucG5nJykKCmdyYXBoX2lsbHVzdHJhdGl2ZV9wbG90cyhtYXhfZGlmZl9zZXQsICcuLi9pbWFnZXMvdGFzay1tYXBfd2l0aF9tYXhfZGlmZl9oaWdobGlnaHRlZC5wbmcnKQoKIyBncmFwaF9pbGx1c3RyYXRpdmVfcGxvdHMobWluX2RpZmZfc2V0LCAnLi4vaW1hZ2VzL3Rhc2stbWFwX3dpdGhfbWluX2RpZmZfaGlnaGxpZ2h0ZWQucG5nJykKYGBgCgpDcmVhdGUgYSBjb29sIDNEIHZlcnNpb24KYGBge3J9CnBsb3RfbHkoCiAgeCA9IGNvbWJpbmVkX2RhdGEkUEMxLAogIHkgPSBjb21iaW5lZF9kYXRhJFBDMiwKICB6ID0gY29tYmluZWRfZGF0YSRQQzMsCiAgdHlwZSA9ICJzY2F0dGVyM2QiLAogIG1vZGUgPSAibWFya2VycyIsICMgY2FuIHVzZSBtb2RlID0gInRleHQiCiAgdGV4dCA9IGNvbWJpbmVkX2RhdGEkdGFzayAsCiAgY29sb3IgPSBjb21iaW5lZF9kYXRhJGNsdXN0ZXIKKQpgYGAKCkNyZWF0ZSBzeW50aGV0aWMgZGVwZW5kZW50IHZhcmlhYmxlIGJhc2VkIG9uIHRoZSBjbHVzdGVycwpgYGB7cn0KdGFza3Nfd2l0aF9kdiA8LSBzdWJzZXQoY29tYmluZWRfZGF0YSwgdGFzayAlaW4lIG1heF9kaWZmX3NldCkgJT4lCiAgbXV0YXRlKAogICAgc3luZXJneSA9IGFzLmZhY3RvcihpZmVsc2UoY2x1c3RlciA9PSAxIHwgY2x1c3RlciA9PSAyLCAxLCAwKSkKICApCmNvbWJpbmVkX2RhdGEgPC0gY29tYmluZWRfZGF0YSAlPiUKICBtdXRhdGUoCiAgICBzeW5lcmd5ID0gYXMuZmFjdG9yKGlmZWxzZShjbHVzdGVyID09IDEgfCBjbHVzdGVyID09IDIsIDEsIDApKQogICkKYGBgCgojIEZpdHRpbmcgYW5kIFZpc3VhbGl6aW5nIE1vZGVscyBmb3IgdGhlIFRhc2sgTWFwLgoKYGBge3J9CnggPC0gY29tYmluZWRfZGF0YSAlPiUgc2VsZWN0KFBDMSwgUEMyLCBzeW5lcmd5LCB0YXNrKQp0cmFpbiA8LSB0YXNrc193aXRoX2R2ICU+JSBzZWxlY3QoUEMxLCBQQzIsIHN5bmVyZ3ksIHRhc2spCm1vZGVsIDwtIHRyYWluICU+JSBzdm0oc3luZXJneSB+IFBDMSArIFBDMiwgZGF0YSA9IC4sIGtlcm5lbCA9ICJsaW5lYXIiKQoKc3ZtcGxvdCA8LSBkZWNpc2lvbnBsb3QobW9kZWwsIHgsIGNsYXNzX3ZhciA9ICJzeW5lcmd5IikgKyAKICBnZW9tX3BvaW50KGRhdGEgPSB0cmFpbiwgYWVzKHggPSBQQzEsIHkgPSBQQzIsIHNoYXBlID0gc3luZXJneSksIGNvbG9yID0gImRhcmtvbGl2ZWdyZWVuMiIsIHNob3cubGVnZW5kID0gRikgKwogIGdlb21fbGFiZWwoZGF0YSA9IHRyYWluLCBhZXMobGFiZWwgPSB0YXNrICksIG51ZGdlX3kgPSAwLjEsIHNpemUgPSAzKSArCiAgbGFicyh0aXRsZSA9ICJTVk0gKExpbmVhciBLZXJuZWwpIikgKwogIHRoZW1lX21pbmltYWwoYmFzZV9zaXplID0gMTIpCgpzdm1wbG90CiAgCmdnc2F2ZSgnc3ZtcGxvdF9zeW50aGV0aWNfZGF0YS5wbmcnLCB3aWR0aCA9IDEyLCBoZWlnaHQgPSA1KQpgYGAKCmBgYHtyfQptb2RlbCA8LSB0cmFpbiAlPiUga25uMyhzeW5lcmd5IH4gUEMxICsgUEMyLCBkYXRhID0gLiwgayA9IDEpCgprbm5wbG90IDwtIGRlY2lzaW9ucGxvdChtb2RlbCwgeCwgY2xhc3NfdmFyID0gInN5bmVyZ3kiKSArCiAgZ2VvbV9wb2ludChkYXRhID0gdHJhaW4sIGFlcyh4ID0gUEMxLCB5ID0gUEMyLCBzaGFwZSA9IHN5bmVyZ3kpLCBjb2xvciA9ICJkYXJrb2xpdmVncmVlbjIiLCBzaG93LmxlZ2VuZCA9IEYpICsKICBnZW9tX2xhYmVsKGRhdGEgPSB0cmFpbiwgYWVzKGxhYmVsID0gdGFzayApLCBudWRnZV95ID0gMC4xLCBzaXplID0gMykgKwogIGxhYnModGl0bGUgPSAia05OICgxIG5laWdoYm9yKSIpICsgCiAgdGhlbWVfbWluaW1hbChiYXNlX3NpemUgPSAxMikKCmtubnBsb3QKICAKZ2dzYXZlKCdrbm5wbG90X3N5bnRoZXRpY19kYXRhLnBuZycsIHdpZHRoID0gMTIsIGhlaWdodCA9IDUpCmBgYA==